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INTRODUCTION 

Under contract to the Army's Night Vision and Electro-Optics Laboratory, 
Westinghouse has been tiestigatine te design, test, and implementation of a 
set of algorithms to perform intelligent. tracking and intelligent target 
homing on FLIR and TV imaery naxenees has been initiated for the develop- 
ment of an intelligent target tracking and homing system which will combine 
target cueing, target signature prediction, and target tracking techniques for 
near zero break lock performance. The intelligent tracker will monitor the 
entire field of view, detect and classify targets, perform multiple target 
tracking, and predict changes in target signature prior to the target's 
entry into an obscuration. The intelligent tracking and homing system will 
also perform target prioritization and critical aimpoint selection. Through 
the use of VLSI/VHSI techniques, the intelligent tracker (with inherent target 
cuer) can be applied to the fully autonomous munition. 

During the first quarter, several meetings and a number of phone 
conversations took place between Westinghouse personnel and John Dehne, 
Peter Raimondi, and Capt. Ben Reischer of NV and EOL. A program plan was 
reviewed, preliminary analytic and computer results were presented, and additional 
program direction was provided. Included in this report are portions of the 
program plan, results of programming the DARPA algorithms, a description of the 
NV & EOL data base, and results of the Westinghouse freeze frame device 
applied to both 525- and 875-line data. The basis for the Tracker Assessment 
Section was provided by Capt. Ben Reischer of NV & EOL. 

Westinghouse personnel participating in this effort include Thomas Willett, 
Program Manager; Dr. John Romanski, John Shipley, Leo Kossa, Tony Cangiliosa, 


Robert Bidney, and Richard Kroupa. Program review and consultation is provided 


by Drs. Glenn Tisdale and Azriel Rosenfeld. 











1,0 SYSTEM DESCRIPTION 





The purpose of this section is to assess the tracking problem in the 
lignt of current techniques, describe the requirements for intelligent tracking 
and homing, examine a system configuration which functionally responds to the 
problem, and allude to some of the appropriate image processing techniques 


which are more fully described in Appendix A. 


1.1 Assessment of Target Tracking Techniques 


This section is an excerpt from some work performed by NV & Eo..? 
The tracking techniques are divided into the categories of: presently fielded 
trackers, state of the art trackers, and future intelligent trackers. Each 


is described in the following paragraphs. 


1.1.1 Presently Fielded Military Trackers 

Presently fielded military trackers typically are of the single mode 
type, most commonly correlation or contrast (centroid of brightness) only. 
Target lock-on is achieved solely by operator command and the object being 
tracked is described merely as a group of illuminated pixels within a fixed or 
user defined track window, Failure of the tracker algorithms to delineate 
the boundaries of the tracked object coupled with poor window sizing (operator 
controlled) permits the introduction of clutter objects within the track 
window. Continuous loss of track in high clutter regions js typical for these 
non-adaptive tracking techniques. 

The correlation tracker attempts window image registration on a frame- 
to-frame basis with repeated update of the reference image through use of the 
previous frame of imagery. Though this methodology compensates for moving 
objects within the track window it also permits clutter objects to influence 
the tracker confidence. Hence a target going behind a large bush may leave 


the tracker locked onto the bush and not the re-emerging target. Target 
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obscurations affect the contrast tracker in much the same fashion. A moving 
target approaching another object causes both signatures to enter the track 
window and hence forces the centroid of brightness to be driven to a point 
between the two objects causing the loss of the originally tracked target. 
Failure of the operator to accurately control the track window size (a 
process known as “gate discipline") greatly exacerbates these breaklock 
conditions and burdens the operator with continually performing manual target 
reacquisition. 

Another shortcoming of presently fielded trackers is caused by the use 
of less than optimal algorithms (e.g., binary correlation) which is required 
to permit implementation in reasonable realtime packages. Even so, the 
fabricated hardware is often bulky, special purpose (mission dependent) hard- 
ware and has a significant power consumption. This often requires a redesign 
of the hardware to compensate for the peculiarities of a new mission scenario, 

Perhaps a more important limitation, however, is the implicit assumption 
that one and only one "target" exists to be tracked in the image. It jis 
this assumption which lies at the heart of the “gate discipline" or correlation 
“reference update" problem. Further, it limits the technology to tracking one 
target at a time although the military necessity of high rate of fire depends 
on simultaneous tracking of multiple targets. Finally, because the taraet is 
known only by its intensity profile, the centroid (hence tracker aimpoint) 
will vary with changing target aspect. This “aimpoint wander" is often one 


of the most significant errors in a tracking system. 


1.1.2 State-of-the-Art Trackers 
The newly emerging state-of-the-art (SOA) target tracking techniques 
incorporate a variety of tracking methodologies in a multimode format. We 


begin to see interaction between the tracker and a controller in a “handshake" 
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manner, An example of this might be a synergistic combination of several 
algorithms for achieving a highly stable tracking system. The algorithms may 
include contrast and edge tracking along with correlation tracking for the 
derivation of target motion and for pixel registered scene tracking during 
breaklock conditions. The use of LSI hardware techniques permits a more 
optimal selection of tracker algorithms for realtime applications and permits 
the incorporation of advanced multimode trackers in terminal munitions and 
man-in-the-lToop RPV scenarios. Microprocessor - based tracking technology 
permits fine tuning of algorithms for specific applications. This is a 
tremendous improvement over fielded (hard-wired) trackers which are not 
adaptable to improved algorithms without hardware modification. 

Unfortunately, SOA trackers stil? encounter severe problems when 
tracking in high clutter environments. For example, the SOA centroid contrast 
tracker now has an adaptive gate, Assume the target being tracked moves into 
an area with other intense objects. Perturbations in tracker confidence force 
the system into a reacquisition mode within this high clutter region. As 
the tracking gate widens to reacquire the target, clutter objects significantly 
influence the tracker algorithms and may drive the system to lock onto a 
clutter item rather than the true target. Additional factors which cause the 
algorithms problems include low target-to-backaround contrast, sun-to-horizon 
angle (shadows, glint), target-sun aspect, backaround texture, etc. These SOA 
trackers still operate only on single targets but have some limited aimpoint 
analysis capability due to delineation of target edges. However, “aimpoint 
wander" jis still a problem. Even though they provide the best tracking schemes 
to date, multimode trackers still Tack the sophistication for application 


into fully autonomous terminal munitions. 


1.1.3 Advanced Intelligent Target Trackers 
Research has been initiated under this contract for the development of 
an intelligent target cueing and target tracking methodologies for near zero 
«4. 
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breaklock performance. The intelligent tracker sought should possess the unique 
capabilities described in the following paragraphs. 

The intelligent tracker should be able to track many targets in the 
sensor field of view simultaneously without duplication »f tracker hardware. 


Tracker/cuer synergism will allow the cuer to continually inform the tracker 
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of all cued objects. The tracker will update its memory to acknowledge the 
existence of a new target or reconfirm the location and track of known targets. 
Since the intelligent tracker works in conjunction with a target cuer, 
identification (or classification) of the tracked object is possible as 
feature information (range, size, shape, etc.) is extracted from the sensed 
scene. This capability (classification) also permits prioritization of the 
tracked multiple targets and critical aimpoint analysis of each. 
Following closely with the Identification capability, the intelligent 
tracker should prioritize all tracked targets within the field of view based 
on apriori knowledge about target type and threat. This analysis will permit 
the operator to always engage the highest threat target first in a multiple 
target scenario. It is impertant to realize that such target prioritization 4 
is by no means a cut and dried procedure. Prioritization must be considered 
hand in hand with threat assessment. Which should be assigned a higher 
priority - a tank or a SA-9 missile? The answer depends on the scenario and 
situation. If the tracker is located in a tank, then an enemy tank would be 
of higher priority than a SA-9 missile. The reverse might be true if the 
tracker is in an AAH,. 
Since the target has been classified and track information is available, 
the intelligent tracker can point to the location of the most vulnerable point 
of the target. Munitions deployment would be directed to that point since a 
hit there would yield the highest probability of kill. 
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The intelligent tracker should track autonomously, automatically 
reacquiring the target as need be, from the time of acquisition until time of 
impact or completed munitions deployment without any human intervention 


required. An extremely important and difficult problem occurs for targets 


entering, leaving, and re-entering the field of view (FOV) as happens in the 


RPV. In such cases, it is very important for the tracker to “remember" 
targets outside the FOV, their classification, direction of travel, and time 
since leaving the FOV. Thus, after the current highest priority target has 
been hit, the tracker would know approximately where to look for the next 
highest priority target even if it was now outside the FOV. 

The problems of track loss and reacquisition, one of the weak points 
of present trackers, should be overcome by the intelligent tracker. Current 
tracker technologies only consider the background immediately surrounding 
the target in track. This approach leads to track loss as the target moves into 
new background regions. The intelligent tracker will contain algorithms 
which monitor a broad context around the target and predict the expected 
variation of the target's signature before it enters the new background 
region. Thus, this advanced approach will allow the tracker to follow the 


target as it crosses the boundary between background types. 








1.2 Intelligent Tracking and Homing Problem 
The reasons for considering some sort of intelligent tracking and 







e 
homing were stated by NV & EOL in the following paragraph, 






“...Present day target traskers and terminal guidance homing 


systems encounter severe degradations in performance when operated 







in environments with high clutter backgrounds. Tne reasons for 






the failure of these techniques is their inability to alter the 






description of the target as it passes into a new background 


region. Instead, these trackers and homers wait unti] the target's 





correlation coefficients diminish and then attempt to reacquire 










the target in a totally undesirable domain". 


NV & Eo? then went on to state the direct technical problem which the 
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Intelligent Tracking and Homing system must attack. 







"...A system of algorithms which can quantitatively identify 






a target and its present background but can acknowledge the approach 


of a new background in the target's path, characterize this new 





region, and intelligently predict the target's signature BEFORE 

(capitals are NVL's) it enters the new domain". 

There are additional features which this set of algorithms must 
possess which further remove it from the realm of present day tracking 
and terminal guidance homing systems, namely* 


"\..must be able to perform such task as target prioritization, 





critical aimpoint selection, and multiple target tracking. ..on 
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visible and infrared imagery". 







These additional requirements force a significant distinction between 






trackers operating in a small window around a pre-assigned target and 





trackers responsible for prioritizing and tracking multiple targets over 
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the entire image. To prioritize, e.g. the system must be capable of 


detection and recognizing targets. 





1.3 System Concept 
The basic parts of the system concept for the Intelligent Tracking 


and Homing System are target prioritization, multiple target tracking, 
critical aimpoint selection, and target signature prediction. These are 


described below. 


1.3.1 Target Prioritization 

Target prioritization is necessary in order to identify as quickly 
as possible that target which presents the greatest threat. A second 
reason for target prioritization is to nullify such countermeasures techniques 
as driving a high priority target among several low priority targets to merge 
their signatures. Target prioritization also eases the need for precise 
pre-launch targeting information obtained under difficult battlefield 
conditions. 

In order to establish an order of importance among potential targets 
in a scene, it is necessary to be able to detect and classify all the 
targets in the scene. This must be done over the entire frame and not 
limited to a small window because of imprecise launch information. This 
must be done with sufficient accuracy and in a short enough time to allow 
an effective change to another target. It is not enough to be able to 
classify targets at such a short range that platform dynamics will not 
accommodate a new course. 

Present trackers do not have the ability to perform their own 
acquisition as is required here, nor do they have the ability to handle a 
full frame of information. Target cuers can perform detection and classi- 


fication based on features extracted from the image such as edges, edge 
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lengths, thresholded shapes, the congruence of edges and thresholded 


shapes, and orientation angles of the edges among others. These features 
are manipulated to form new entities, e.g. perimeter length, area, 
length to width ratios, etc. which are used to reach a classification 
decision concerning some object in the image. These classification entities 
are part of the resident program and are designed to handle a wide variety 
of target aspects, ranges, and approach angles. Under the expected 
clutter conditions in which the intelligent tracking system is expected 
to operate, these features must be modified on-the-fly for effective target 
classification and prioritization. Consider the previous example where a 
high priority target drives among a group of lower priority targets and 
only a portion of it is visible. Modification of the classification criteria 
will be discussed more fully in the Target Signature Prediction Section. 

Target cuers have the ability to handle a ful) frame, 525-lines, of 
imagery at speeds of 3-10 frames per second and the capability to detect and 
classify up to 30 targets per frame. These 30 targets, their classification, 
and position within the frame are maintained in a file. Then a simple 
form of prioritization consists of reordering the file, after every cued 
frame, based on target classification. 

However, the cuers do not provide data at speeds necessary to handle 
a platform control system, only the presently fielded and state-of-the-art 
trackers can do this. So the tracker is directed to follow the highest 
priority target provided by the cuer. The combination of a target cuer 
and frame-to-frame tracker provides the ability to survey the entire frame, 
detect and classify targets therein, and provide frame-to-frame tracking 
information on the highest priority target. Should a higher priority target 
be discovered, during the course of tracking some other target, the cuer 


would direct the frame-to-frame tracker to the new target and continuous 
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tracking data would be obtained from it. Further, the frame-to- frame 


tracker would be provided a target image by the cuer. 


1.3.2 Multiple Target Tracking 

To track more than one target, frame-to-frame trackers generally 
require a duplicate set of hardware for each additional target. In auto- 
nomous munitions or RPV's where volume is at a premium, this is not a 
satisfactory approach. Alternately, the combination of a cuer and frame- 


to-frame tracker provides continous tracking on the highest priority target 


over the entire image. In this fashion, 30 targets per second could be tracked 
simultaneously with the same piece of hardware. The latest positions 

of the other targets are available from the target list. From this list, 
target tracks of other than the highest priority target, can be computed 

even when they leave the image. Visual contact with these targets can be 
re-established by following these computed tracks. These additional tracks 

are of sufficient accuracy to allow the frame-to-frame tracker to be 


directed to anyone of them and continuous tracking immediately initiated. 


| 
l 
| 
| 
| and tracking data at the rate of 3-10 track points per second on the others 
| 
| 
| 
1.3.3 Critical Aimpoint Selection 
Frame-to-frame trackers have the ability to perform aimpoint based on a known 
target type. That is, the cuer again must provide the classification for 
the tracker. This algorithm is particularly appropriate at long range 
because it is based on the exterior shape. As the range closes, this 
algorithm becomes suspect and the additional interior shape information 
provided to the cuer is more appropriate. Assuming that the features 
extracted are the same as those used at long range by the cuer, the problem 


becomes one of trying to classify interior portions of a target based on 


the same features set, but different classification entities. For example, 
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it is desirable to be able to use edges, edge lengths, thresholded shapes 
etc. to be able to classify treads and turrets of tanks in order to provide 
accurate close-in homing. This is a classic problem in all homing systems, 
e.g. torpedoes which do not have a high resolution sensor appropriate 

for close-in homing. Again, since high data rates are imperative for 
close-in homing, the frame-to-frame tracxer must provide the data input to 
the munition platform control system on a continuous basis. Here, assuming 
the turret happens to be a desirable aimpoint, the cuer isolates the turret 
as an entity. The frame-to-frame tracker can then be directed to initiate 
centroid tracking on the turret. This is the same procedure used at long 
range on the external target shape. The cuer and the tracker work to- 
gether for, in the case where the target becomes partially obscured, the 
centroid can be calculated based on target classification and the visible 
portion, rather than the visible portion, alone. The aimpoint selection 
based on interior shape detail also provide a check on the long range 
classification algorithms, for the interior detail] must be consistent with 


long range classification in the case of a correct classification. 


1.3.4 Target Signature Prediction 
The intelligent tracking and homing system is required to operate 


in a high clutter environment in which the target may be occluded in a 


number of different ways. These occlusions will not only hinder classification 


and aimpoint porcesses, but they will act to break track and inhibit 


detections. At the feature extraction level, not all of the features wil] 


be available once some portion of the target is occluded. For example, consider 


the case of a dark target travelling on a light road where the shoulders 
appear as dark background. As long as the target stays in the middle of the 
road, there is no difficulty in tracking the target. However, as the target 
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approaches the shoulder, the front part of the target will merge with the 


side of the road (See Figure 1-1) and the target cannot be segmented from 


Sener 


Figure 1-1. Obscuration 






the shoulder. If, for example, we demanded four perimeter edges for a 
target detection, no detection would be made because one of the edges is not 
present. If we had modified the target detection criteria, on-the-fly, to 
three perimeter edges, then a target detection would be achieved. To have 
successfully modified the target detection criteria, we would have had to 
do a number of things, namely; 

1. detect target in middle of road 

2. estimate direction of travel 

3. note background in path of target 

4. compare potential background with target 

5. note conflicts with present target detection criteria 

6. modify to allow target detection 

To perform these steps in the example described, we might have compared 
the gray levels of the target with those of the shoulder of the road. If 
some difference was found we might change a threshold to preserve the 
difference and stay with the four perimeter edge criteria. If no difference 
was found, we might modify the criterion and use only three edges, this 
calls for a similar modification in the classification criteria. Another 
possibility is that the presence of the target against the background changes 
the gray scale pattern over the merged area, and this pattern is different 


from other parts of the background. For example, a light target passing 


-12- 


- ame . aa : 
; 
a ii iia acmsaaaicuiaiiaiiieaeaidaaaailll 





ensusrammasemamaanaunst 


ve eee 








behind a dense dark clump, changes the gray scale value over the dark 
clump at the pixel level. The latter is an example where the initial 
merging acts to confirm, deny, or establish a different target signature. 
In any event, the cuer is looking at the background ahead of the target 
and attempting to predict the change in target signature before it enters 
the new background. Having determined a new target shape because of an 
approaching obstruction, the cuer must inform the frame-to-frame tracker 
and pass the new shape to the tracker. Similarly, when the cuer is ready 
to analyze a new frame, the tracker must tell the cuer where the target 
was seen on the previous frame and pass on some characteristics which are 
relevant to the cuer's analysis. In Sections 2.1 and 2.3, we shall 
discuss some of the more difficult tracking scenarios found in the NV & EOL 


data base and their location in the data base. 


1.4 System Configuration 

Figure 1-2 is the system block diagram for the Intelligent Tracking 
and Homing system. The preprocessor, segment, feature extraction, and 
classification functions may be considered as portions of a cuer. The cuer 
is the controlling element. It provides for initial segmentation of the 
image, directs the tracker to follow that detected target which exhibits 
the highest initial priority, based upon mission data (such as probability 
of occurrence in relation to scene description). Segmented objects of lower 
priority are stored for later consideration. The cuer then performs 
classification of segmented objects based upon features extracted from the 
preprocessed image data. As the tracker follows the priority target, these 
features will be modified and updated by the signature predictor, which 
will make use of the high speed tracker position data as well as the 
preprocessed image data. The signature predictor will supply modified image 
data to the feature extractor for use in obtaining an improved feature set 


for classification. 
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Figure 1-2 is repeated as Figure 1-3, additionally the signature 
prediction fucntions are shown individually. This breakout should be 
regarded as a “straw-man" system in that it serves as a starting concept 
for signature prediction. Successive registrations by the tracker serve 
as a basis for a least squares fit for track prediction; a second order, 
recursive implementation using 5 or 6 points is a reasonable first approx- 
imation. Then portions of the background which lie in the path of the 
target are placed in memory and the target image is superposed and cued. 
From this we obtain an estimate of target image degradation and adapt the | 
cuer functions accordingly. To add more power to the system, we shall | 
perform change detection on the target as it enters the background and 
assemble the target from these features. This accomplishes several objectives; 
it serves as a check on the predicted image degradation, it is useful in 
the case where the target moves behind a screen of trees, and it can be 
used to assemble the target for aimpoint selection. The final step in 
signature prediction is to isolate the appropriate portion of the target for 
aimpoint selection. Hence, we predict image degradation ahead of time, 
modify the expected feature set, check on the prediction, modify it if 
necessary, and isolate the target portion required for aimpoint. The 
purpose of the tracker is to provide position data at the field or frame 
rate of the sensor, and thereby incur a ation time lag to the guidance 
system. The cuer, as presently implemented, will operate at one-third to 
one-tenth the frame rate. However, if the cuer can be operated at the 
higher speeds, and with adequate positional accuracy, the tracking function 
could be accomplished by the cuer. 

The preprocessor is shown separate because it provides inputs to the 
cuer and tracker. Appropriate enhancement and prefiltering functions are 


carried out of the sensor data rate. 
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Segmentation can be accomplished by thresholding and level slicing, 
as with the Maryland algorithms, by gradient following, as with AUTO-Q, or 
by an appropriate combination of these. 4 
Features may consist of such measurements as lengths, areas, mements ! 
on the segmented areas, or measurements made on extracted edges as with f 
AUTO-Q. The edges which offer end-point position as well as polarity : 
(relationship of light and dark sides), provide a natural entry into the 
problem of aimpoint selection. If sufficient resolution is available, target 
details can be outlined and identified. 
Classification is based upon stored reference features for the target 
of interest. When adequate resolution becomes available, aimpoint selection 
begins. 
Appropriate mission data can be stored, including information regarding 
expected targets, background context, and expected target sizes as a 
function of mission dynamics. Prioritization is carried out on the basis 
of all stored available target information, beginning with initial detection, 


and moving up through classification and aimpoint selection. 
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2.0 DATA BASE 

The purpose of this section is to describe the more difficult 
tracking scenarios as found on the NV & EOL Data Base, to describe the 
Data Base itself, and to classify the Data Base in terms of the difficult 
tracking scenarios. These difficult scenarios were alluded to in the Target 
Signature Prediction Section and will be used to design the Signature 


Predictor Algorithms for the Intelligent Tracker. 


2.1 Difficult Tracking Scenarios 
These scenarios describe cases where the target becomes occluded 


such that a portion of the target is not seen. Ina few cases the target 
almost completely disappears and then appears again at a different position 


not predicted by the last seen heading. 


2.1.1 New Background 

In these cases the background is characterized as relatively 
homogeneous but of similar gray scale to that of the target. Also the back- 
ground is something other than another target. For example, a dark target 
is moving onto a dark road, a dark target is moving along the dark shoulder 
of a road, or moving along a muddy path in an open field. Another case 
occurs when a APC turns and moves away from the sensor thus obscuring the 
light front and its gray level now closely matches that of the background. 
Similar remarks can be applied to light targets against light backgrounds. 


2.1.2 Target Shape Merging 

This is a case where the target moves behind or among a group of 
lower priority targets. In this case, part of the high priority target 
is occluded by the presence of the other targets. This is particular 
difficult at long range and low platform altitudes. 
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2.1.3 Target Partly Occluded by an Obstruction 

In this case, the target is occluded by a natural obstruction such 
as a tree, hill, or bend in a road. In all these cases, a portion of the target 
remains in view, and before that portion disappears, another portion re-appears. 
An example is a tank moving behind a tree where the front part disappears, 
the rear is visible, and before the rear portion completely disappears, the 


front portion reappears. In these cases, the obstructions are vertical. 


2.1.4 Target Partly Occluded by Terrain Features 

Here, the target is obscured by low lying shrubs, bushes and the 
like. Other examples include small hills which hide the lower parts of the 
target. The most frequent example of this type of obscuration is a target 


moving in scurb terrain or a clearing in a wooded area. 


2.1.5 Target Almost Completely Occluded 

The target moves behind a screen of trees where the target appears 
as one or more bright spots moving randomly in the general direction of 
travel. The target may be moving down a road, but the geometry is such that 


the target is seen through a screen of trees. Another example is a target 


moving through a thick woods where there are scattered openings. 
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2.2 NV & EOL Data Base 

The purpose of this section is to describe the NVL FLIR data base 
supplied under this contract. For the first pass through the data base, 
we were interested in moving targets with some amount of obscuration and 
long range targets. Portions of the tapes where the targets are larger ' 
will be of interest in the future work under this contract but not at this 
point. Description of tape segments will have more detail if they fall within 
the above criteria. We expect this descriptive log on the other segments 
to expand as the work focuses on them. The index number for each tape 
segment refers to the length meter on the Westel tape recorder. This may 
vary for the reader's case depending on how much tape is used in the threading 
Operation. Nevertheless, the index should be within + 25 of the meter 


reading on the reader's 875-line tape machine. 


2.2.1 Tape "11/10/77 Runs 1,2,3, etc. 


Tape Position 0000-0370 


Helicopter approaches three targets Road 

which are stationary and not obscured; APC TRUCK JEEP 

aman is standing on the APC. The . : " 

APC presents a front aspect, and Direction 
of Flight 


both the jeep and truck present side 


aspects. 


Tape Position 370-670 
A repeat of the above target geometry 
with the targets seen from a longer 
range and higher altitude; the man 


is not present, 
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Tape Position 710-1220 


A repeat of the 370 run geometry 


pre sersneresar os 


with a side aspect presented by 


both the APC and truck and a front 


nae 


aspect for the jeep. 


Tape Position 1265-1580 


APC circles other two stationary 


pres a pa LIE TT 


targets which are not obscured. PC 

The dots (+) on the APC track Teck Pn 
indicate positions where the APC 

shape merges with that of the ‘ 
stationary target. The first Helicopter 


Direction of 
Flight 


several "target merges" occur 
when the targets are 4-8 lines high; 
the latter merges occur for target 


sizes in excess of 15 lines. 


g Tape Position 1780-1950 oo 
ii APC circles other two stationary | 
Jeep APC 
° 


targets and then performs a circular 


is maneuver between them. Dots indicate 
positions where the APC shape merges 
with that of the stationary target. ‘\ 
APC signature is faint along right Helicopter 
Direction of 


side of initial circle. This 
Flight 

sequence was taken at longer 

initial range than 1265-1580. "X" 

indicates position where APC exceeds 


4-8 lines. 
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Tape Position 2032-2400 
Truck initially positioned behind 
jeep and signatures are merged at 
location indicated by dot (+). 


Initial size in the 4-8 line category. 


Tape Position 2500-2800 
Targets are stationary until 
helicopter approaches to approximately 
8 lines of target, then APC moves for- 
ward. APC is initially in a dark back- 


ground and moves into a lighter one. 


Tape Position 2850-3180 
Three targets are initially stationary, 
with two of them on the edge of a dark 
background which appears to be a muddy 
portion of an open field. The jeep moves 
next to the APC causing the signatures to 
merge (+), and then moves ahead a short 
distance where the signatures are distinct. 
Targets are very small during first portion 


of the sequence. 
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Tape Position 3200-3660 
The three targets are initially stationary 
and smal?; there are a few frames where 
the operator switches from the wide 
field of view to the narrow one. When 
the targets are stil) Jess than 8-10 


lines, they move together toward the 





helicopter. 


Tape Position 3700-4000 


Helicopter 


Targets initially at long range and 


closely spaced; the two APC signatures oe 
are merged at position marked by (+). apc? O« Jeep 
APC moves through dark and light back- 
ground and sequence ends with a close 
up of front quarter of moving APC. 
T uchidieeee 
Tape Position 4100-4300 
Initially a jeep is seen from the side — 





and then rear aspect as it moves along 


; Jeep 
a road. Then an APC is seen at close 


range from the rear aspect and followed sik ea 
APC 
as it moves along the shoulder of the * o 
“ 
road. eg + 
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Tape Position 4300 


First the jeep is seen at long range 







by the helicopter at position (X). 






The jeep is moving on the road and the 


jeep signature appears to be faint 






and moving through light and dark 










patches of background. The APC is 







then seen as it moves along the 






shoulder of the road and the APC 


signature merges into light and dark 










background patches as it moves. The 






sequence ends with narrow field Helicopter 





images of the APC seen from quarter 









The truck, seen 






rear and rear aspect. 






briefly in several frames, comes 






around the bend in the road where the 


jeep was first seen and both truck 







and APC appear close-up, in the image. 





Tape Position 4760 





Side and rear aspect images of APC 


and truck as seen from medium and — 


close ranges. Several times the o— 
APC 





target outlines merged and then drew 






apart. Both targets are in the image wooo? 
Helicopter 






continuously until a final close-up 







shot of the APC. 





Targets seem to be 





running at high speed. 





eran 


Tape Position 5000 


An apparent mirror image of 4760 with Tank 
Tank 
the APC and truck moving in the opposite % 
APC 
direction and the closest target to the se 
helicopter is the truck; sensor comes uick 


upon two stationary tanks at 5270 during 


course of tracking truck and APC. Pen 
Helicopter 


Tape Position 5365 


The APC in the lower part of the figure 
is seen from the side, moving fast, at 
medium range. It disappears behind the 
screen of trees, and is seen again when 
the helicopter gains altitude. Then 
the helicopter switches to the other 
APC, moving more slowly. Side, 


quarter-rear, rear, and quarter-front 





aspects are seen at close ranges. 


2.2.2 Tape "11/11/77 1030 hours" 


Tape Position 380 Road 
There are five closely-spaced, ape TN 
stationary targets seen beginning at long oo00 0 
range and closing. At long range the 
right-most APC and truck signatures are 
merged and there appears to be only four | 
targets. Helicopter 
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Tape Position 800 
Truck Jeep 
There are five targets which are APC APC APC 
stationary and pointed toward the 00000 


helicopter. At long range the 


images of the truck and APC to its [ 
H 


right appear merged. elicopter 


Tape Position 1000 
The same as 800 with helicopter range, 


aspect, and altitude changes. 


Tape Position 1320 
The same target geometry as the above 
and the helicopter approach angle is 
such that the merged truck and APC 
images do not separate until the 


range is very short. 


Tape Position 1730-2300 
Repeat of target geometry but initial range 
is longer. Helicopter approach angle - ae on 
is such that hidden APC is seen more APC APC 
clearly but merges with the truck as 
approach continues. The left~most APC Helicopter 
has the hatch down and the APC to its 


right has the hatch up. 
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Tape Position 2430 
At long range, the APC images appear 


merged, similarly the jeep and truck 
Road APC 
Oapc Jeep 
o.6|068 


images appear merged. There is a 
muddy path between the four stationary 


vehicles. The moving APC, which appears 


© °o 
dark with a light front, merges with the APC Truck 


other targets, as it passes among them, and 
the muddy path. The dots (@) show the 
merge positions of the stationary 

target with the moving APC. Front edges 
are lost, and target bottoms are lost. 

The jeep and truck are very light targets; 
the moving APC is darker than the 


stationary APC's. 


Tape Position 3100 
Same target geometry as 2430, but the 
helicopter altitude appears to be lower 
so the occluding targets are seen as closer 
together. In this case, the moving APC 
crosses the road twice before passing 
through the targets again. The dark 
patches on the road and shoulder provide 


obscuration for the dark APC. 





Tape Position 3700 Road 


Same geometry for the stationary targets 






as 3600 but this time APC starts from oO 





Mp Oo O 
APC Truck 


across the road. The dark APC must 


CNTY RT TAREE Sc REESE EMORY Evra 


pass over the dark road and over the 


muddy ground before reaching the targets. 


antares ta aae eee 


Tape Position 4380 f 


Repeat of same geometry as 3700, but 











range is much shorter before APC APC Jeep 
: , APC O Oo 
i begins moving. Closing helicopter re) 
is seen as representing a munition, age Truck 
and APC just reaches first set of 
vehicles when impact would have Helicopter 
| occurred. 
i Tape Position 4855 
| 
a; | Same stationary target geometry as APC APC Jeep 
: Oo 
above, but moving APC starts from right 0 . 
side of the road. The closing helicopter APC Truck 
flies over the moving APC as it emerges 
the truck and jeep. 
Helicopter 
Tape Position 5330 
Same stationary target geometry as ie = 
above, and moving APC starts from on APC “a— O 
the road; the APC emerges from the first we Truck 
set of targets as the helicopter flies over. 
elicopter 
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i, 2.2.3 Tape "#4" 
Tape Position 240 
APC moving from woods into clearing. 
Woods 
| The woods at the beginning of the 
run and the trees at the end partially ‘ 
occlud the moving APC. The movin 
. ° Jeep oO : 
APC's image merges with that of the 9 © Truck i 
PC 
jeep and stationary APC at the positions APC 
i 
indicated by dots (@). 1 Trees i 


Tape Position 500 Helicopter 
Same stationary target geometry as 
240 but moving APC starts out as 
dark target against dark road. The 
moving APC is 4-8 lines high at the 
position denoted by an "X". The moving 
APC is occluded by trees as it passes 
the truck and moves back toward the road. 


Initial helicopter ranges are much 





longer than 240. The image of the 


' moving APC merges with the images of each —— 
i 
: of the stationary targets successively 

| at the positions shown by the dots. Helicopter 


| 
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Tape Position 803 
Repeat of 500 except APC moves 
more rapidly so that target sizes 
are smaller at run completion as heli- 
copter closes. Also end of run shows 
APC traversing the road longer. APC 
starts from edge of the road instead 


of the road itself. 


Tape Position 1080 
Repeat of previous run and target 
geometries except that moving APC 
starts from the road, Long range is 
maintained throughout most of this 
run. Occlusions seen with trees shield-~ 
ing the moving APC from view and APC shape 
merging with that of the stationary targets. 
Also, as in the other runs in this series, 


500 and 803, the APC moves along the edge 


of a woods in the top part of the run. 


Tape Position 1380 
A repeat of 1080 except done at longer 
range, and the moving APC stops when it 
is between the APC and truck so that 


there is merging of the target shapes. 





Oe We 
ot 


Tape Position 1610 

A short set of frames showing the 

[ stationary targets in the clearing 

| arrayed in the same geometry as 580, 


| 803, 1080, and 1380. 








SP 


Tape Position 1870 


Run starts at long range and high al- 


titude such that targets are very smal], 


Targets are stationary for about half 
the run and then the APC moves from 

the edge of the woods to between the 
APC and truck. Obscurations are caused 
by ground cover and shape merging 

with APC at position indicated by 

dot (¢). 


Tape Position 2200 


Sensor operator seemed to be scanning 


for entire run length. 


Tape Position 2800 


Same stationary target geometry as 1870. 


Moving APC position is further back in 
woods, so obscurations sometimes mask 
the entire APC. Helicopter closes 


rapidly and at such an angle that shape 


merging occurs with jeep. 





APC 


Jeep O 


° Ol ruck 
APC 


Helicopter 


APC 


Woods+> 


Jeep © 


Oyruck 
Rec 


Helicopter 


a lc ll ct 





a short distance up the road and stops; 
Jeep then moves up to position slightly 
behind APC so that their shapes merge. 
Then as truck moves to merge with the 
APC and jeep, the APC moves forward and 
then into woods. The truck follows, 
passing the jeep, the APC in the woods. 
Trees and brush occlude al) or parts of 
the targets as the move along the road 
and in the woods. The jeep moves up 


the road at the end of the run. 


Tape Position 3700 
Movement of four target through woods 
shown in this run. Targets first appear 
as a single dot, then appearing and 
disappearing dots, and the four targets 


emerge from the woods at the end of the 


sequence, 


| 
Tape Position 3090 
\ 
Targets are initially grouped, with 
shapes merging, as shown. APC moves 
| 
® 
i 
' 
, 
: 
; 
| 





APC 
Jeep 


Truc 


aes 


rged Targets 





r 


* ee Woods 


x 
Truck © © Jeep 


© APC 
apc © 


Tape Position 4400 
Same target geometry as 3700 except 


targets are initially closer to the 


nS 


clearing, helicopter range and al- 
titude are less. This means that the 
initial portions of the targets seen 
are larger, and complete occulsions 
are of short duration and near the 
beginning of the run. Many examples 


of partial obscuration, 


L. Tape Position 4850 

Repeat of 4400 except helicopter is 
closing faster and altitude is higher. 
Helicopter closing angle is such that 
complete occulsions are obtained with 


‘ fairly large targets. 


t Tape Position 5300 

Same target geometry as 4850. 

Helicopter starts at closer range and 
targets are always, at least, partially 

visible. Examples of partial obscur- 


ation and shape merging. 
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Tape Position 5650 


Same target geometry as 5300 except 


pammmeed 


target start from further back in 

the woods. This provides examples of 
complete obscuration as well as 
partial. Also examples of shape 
merging. Targets move out of woods, 


across clearing, and into woods on 


a 


the other side of the clearing before 


helicopter flies over. 


Tape Position 6000 


Single APC, at fairly high speed, 


appears as a dot, partially obscured, 
then emerges from the woods. The APC 
then turns in the direction of in- 
coming helicopter and moves among 

trees on the other side of the clearing. 


Helicopter 
Tape Position 6270 Approach 


Three targets, with merged shapes, are 
seen moving through woods as alternating 
white dots, then partial obscuration, 
and as complete targets at the end of 


the run. 








2.2.4 Tape "11/14/77" 
This tape consists of stationary targets 
throughout which were in a clearing. Since 
there was no target movement, this tape was 


not analyzed for tracking purposes. 


2.2.5 Tape "#5 11/15/77" 
Tape Position 330 
Helicopter approaches stationary 
targets in trees along border of 
woods. Tank appears as dark 


target with light turret. 


Tape Position 600 
Same target geometry as above, but 


helicopter madified the appraoch angles. 


Tape Position 840 
Same target geometry as above but 
helicopter approach is at greater al- 


titude and approach angle of 600. 


Tape Position 1100 
Same target geometry as above with 


helicopter starting run at longer 


range. 





cach CRRA Rtn tenet tesa us ee ee : 





Pick Up 
Truck 





Helicopter 


Pick Up 
Truck PC 


Truck Truck 


AP 
Helicopter 





Tape Position 1395 
Same geometry as 600 with long range 
images and much shape merging as ob- 


scuration. Approach speed appears slower. 






Tape Position 1750 Road 
Targets are stationary in open field. i ea ad } 
The tank and muddy portions of the field TankO o 9° 
appear dark. The truck and jeep a eo 
appear light, and the APC's appear as i 
gray targets. 1 Helicopter 
Tape Position 2030 | 
Same initial geometry as 2030, but tank 
moves across road and through the two Truck APC 
pair of stationary targets. Tank under- Tank Oo O 






Oo Oo 


goes obscuration with backgrounds on 
Jeep APC 





left and right sides of road and then 


shape merging with each pair of targets. 
Helicopter 


Tape Position 2300 Tank 
Tank starts partially in mud to the : a Truck APC 
right of road and passes between pairs = - 
of targets with shape merging. Pi 9. 


Helicopter 





ee 


Tape Position 2560 Tank 


< Truck APC 
°o. 60 


o Oo 
Jeep APC 


Repeat of 2300 except at longer range 





and slower helicopter approach speed 





resulting in smaller targets. Muddy 


portion of field to the right of targets 


provides obscuration of tank with f 

background. Helicopter 
Tape Position 2910 Tank 

Repeat of 2560 but now muddy path has . Truck APC 

been worn between the target pairs so O 

that bottom of tank over entire run is Jeep R 

obscured. t 


é Helicopter 
Tape Position 3180 


Tank starts from left side of road; - 
helicopter is at long range, initially. 
Helicopter flies over when tank is 


between the two APC's. Bottom of tank ts 0 





only visible at the beginning of the run. 


Tape Position 3510 Helicopter 
Same geometry as 318 but closer initial 


range; helicopter flies over when tank 


has just cleared the APC's. 











rae 


ms Breen 
- 
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Tape Position 3780 
Rear aspect, at close range, of truck and 
tank moving along road. The truck continues 
on after going around corner, but tank 


stops at (x) position and a portion of it 


Truck 
is seen as helicopter approaches, 
Tank 
Tape Position 3910 
At tape position 3925, one small dot appears. "nis cideien 


Later, at 4060 dot reappears, larger in 
size, but with a substantial amount of 
obscuration. A second target appears and 
both are partially obscured by a screen 


of trees as they move along a road. The 





second target is a tank. 


Helicopter 


Tape Position 4250 


Tank 


Dark tank is initially positioned on a 
dark road; the tank then moves off the Wood 


road into the wooded area. The tank 





begins moving when the helicopter has 
closed to approximately medium range. learing 


oad 


i 


Helicopter 
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Tank Position 4580 


Close up view of tank as seen through 
a screen of trees providing partial 


obscuration; side and rear aspects. 


Tank Position 4700 


close up view of tank as seen through & 
screen of trees providing full and then 


partial obscuration; front view. 


Tank Position 4880 


Dark tank on a dark road and partially 


obscured by trees at the road inter- 


of the Figure. Vehicle following tank 


is a jeep. Initial range is long. 


Tank Position 5080 


Tank, initially moving on road, moves 


tank are obscured. Another vehicle, 





a jeep, is moving ahead of tank and 


| stays on the road. The tank passes the 


jeep in the last part of the run. 
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section and atong the road at the bottom 


off onto shoulder. Bottom and treads of 





«Trees 





© Tank 


Tank * 
Jeep F 
0 
Tank 


Trees 





Scena 


Tank 


Jeep 
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Tape Position 5280 


2.2.6 Tape "#2 11/11/77" Helicopter 
Tape Position 100 


APC 
shapes are merged and the three APC Jeep O aoe 
shapes are merged at long range. The Truck — 
jeep and truck move together across 
the road where background obscuration 
occurs with the light targets against Jeep apc 
the light road. Once across the road, iil ae 
the truck slows and the jeep moves freck o_, 
ahead. b. 
Tape Position 390 Helicopter 
Same initial position for the jeep and 
truck but the three APC's are on the road. 
Further, the jeep and truck stay together JeepO \Q-—* 
this time as do the three APC's when all sruct? —» 


The jeep is moving on the road and the 


tank on the shoulder. Both targets 


appear at close range as they go around 
the curve simultaneously. Run finishes es.) i 


with close range images of the tank seen © Jeep 


x 


at rear aspect. 


The initial target positions are shown 


in figure a. The jeep and truck 


five targets begin moving from left to 





right. 1 Helicopter 
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Tape Position 865 
Same geometry as 390 except the positions 
of the jeep and truck are reversed. Most 
of the run occurs at long range because 
the initial helicopter range is long and 
the closing speed is slow. Obscurations 
are shape merging and background when 


the light targets are on the road. 


Tape Position 1350 
There are five targets placed in the 
trees along the border of a woods. The 
targets are stationary. Initial range 
is long and there is shape merging; the 
right most APC is difficult to distinguish 


at long range. 


Tape Position 1700 
Same target geometry as 1350 except 
approach angle of the helicopter has 
moved to the left. Obscuration still 


occurs among the same three targets. 


APC 


APC O Truck 


APC 


Jee 


Helicopter 





2.3 Classification of NV & EOL Data Base into Difficult Tracking Scenarios 
According to Section 2.1, the difficult tracking scenarios were 


listed and discussed; they are repeated here and labelled with a letter which 
refers to them from the table of NV & EOL data base tapes. 

A. New Background 

Bl. Target Partially Occluded by an Obstruction 


B2. Target Partially Occiuded by Terrain Features 


C. Target Almost Completely Occluded (Isolated Area Visible) 


D. Target Shape Merges with Another Target 


Examples of these obscurations are shown in the images of Reference 3. 





TABLE 3-1 
Difficult Tracking Scenarios in NV & EOL Data Base 


| TAPE OBSCURATIONS. 
A Bl B2 c D 
Tape 11/10/77 1265-1280 1265 
Runs 1,2,3 1780-1950 1780 | 
2032 2032 | 
2500 2850 . 
3700 3700 
4300 
4760 4760 
5000 
Tape 11/11/77 2430 2430 
| 1030 hours 3100 3100 
3700 3700 
4380 4380 
4850 4850 
5330 5330 
L. Tape #4 240 240 240 
1 500 500 500 500 
830 830 830 830 
1080 1080 —-:1080 1080 
1380 1380-1380 1380 


1870 


2800 





TAPE 


Tape #4 
(Continued 


TABLE 3-1 


Difficult Tracking Scenarios in NV & EOL Data Base 


(Continued) 
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OBSCURATIONS 

Bl B2 
3090 3090 
3700 3700 
4400 4400 
4850 4850 
5300 5300 
5650 5650 
6000 

6270 


3090 
3700 


4850 


5650 
6000 


3090 
3700 
4400 


5300 
5650 


6270 


TABLE 3-2 


Difficult Tracking Scenarios in NV & EOL Data Base 
OBSCURATI ONS 


Bl B2 





3.0 PRELIMINARY RESULTS 

The purpose of the section is to describe that portion of the image 
processing laboratory at Westinghouse where this contract work is being 
performed, to describe the results of programming the University of 
Maryland algorithms, and describe the test results of the Westinghouse 
digital freeze frame device. These items are Westinghouse obligations 
which were discharged during the first quarter of this contract. 


3.1 Westinghouse Image Processing Laborator 





We shall be using the system shown in Figure 3-1 to analyze the data 


base supplied by NVL. 






UNIVAC 
1210 





875 LINE 
HAZELTINE 
OISPLAY 


Figure 3-1. Laboratory Set Up 


Two video tape recorders are being used, a 525 line % inch tape system 

and an 875 line, GFE, 1 inch tape system. A 125x125 window is snatched by the 
Westinghouse freeze frame device with a 17 MHz oscillator, digitized and 

put in a (RAM) Random Access Memory. The RAM is then written (6 bits 

of data) onto a magnetic tape. The freeze frame device is clocked so 

that an arbitrary sequence of frames e.g. consecutive or every nth 


frame can be obtained. The magnetic tape is then read by a Univac 


1110 wherein the intelligent tracking algorithms reside. These algorithms 





Bee 


are written in Fortran so that change in the algorithms can be accomplished 
easily in a high order language. After processing the images with the 
algorithms, a hard copy is available of the resulting images. The processed 
images can also be put on a magnetic tape and processed by the Westinghouse 
Simulation Laboratory where Polaroid photographs of the displayed processed 
images are obtained. 

In a typical run, a section of video tape encompassing 750 
windows and representing 25 seconds is scanned. From the target velocities 
seen on the NVL data base, every tenth frame is a decent representation. 
The representation we are seeking is before the obscuration, during the 
obscuration, and after the obscuration. From the 75 frames obtained, some 
of the frames will have no information, because of sensor and platform 
motion. We will end up with approximately 20 good frames, some depicting 
each of the above conditions. We feel that we can synthesize the intelligent 
tracking algorithms from such a collection. Having programmed the equations, 
we can then go back over these sequences and test the equations against 
every frame to cover the gaps. 
3.2 DARPA Algorithms 

The DARPA algorithms developed under the SMART sensor contract were 
described in Appendix A of this report. These algorithms are now programmed 
in Fortran on the Univac 1110. Figure 3-2 is a scene from the 525 line NVL 
data base. This frame has been taken from a video tape, digitized, and 
placed on magnetic tape and then the Univac 1110 disc for processing. After 
processing, the hard copy was given to NVL at a January meeting and the 
photographs of digital CRT display are shown in Figures 3-2, 3-3, and 3-4, 
Figure 3-2 is the scene before processing; Figure 3-3 is the image after 
median filtering; and Figure 3-4 shows the superposition of the connected 


components and non maximum suppression algorithms. 
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Fiaqure 3-4, Superslice Alaorithms Results 


Note that the taraets were only two or three lines high and the Median 


Filter was set for 5x5 so that the three targets to the right of the road 


appear as a single target on the median filtered image. Whereas on the original 


image, one can just barely discern the separations between the three. In 


Figure 3-3, the highlights on the target exteriors show the coincidence 
between perimeter points of the connected components algorithm and the 


gradients left by the non maximum suppression algorithm. 


Figure 3-5 shows the output of the Westinghouse Auto Q system at 


roughly the same stage of processing. At this point, no attempt is made 


to take either set of algorithms any further. The thrust is to collect an 


obscuration data base and then apply the alaorithms to it. 
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3.4 875 Line Data 


. Figure 3-11 and 3-12 show results of snatching 875 line video as shown on 


the Hazeltine Terminal in the lab. The oscillator was 12 MHz for these cases 





and was then updated to 17 MHz to avoid aliasing. The data came from FLIR 
Tape "11/7/77, Practice Run, Lothads," at tape position 2500. 
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Figure 3-11. 875 Frame #1 
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APPENDIX A 


A.1 APPLICABLE IMAGE PROCESSING TECHNIQUES 


A.1.1 Segmentation Techniques 


One set of segmentation techniques utilized for this contract was 
developed by University of Maryland4 under the auspices of DARPA and NVL. 


The system flowchart is shown in Figure 
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Figure A-1. System Flow Chart 
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In general, the Median Filter acts to suppress noise. The Gradient Operator 
extracts edges which are then thinned by the Non-Maximum Suppression Algorithm. 
At the same time, a set of gray levels is determined and the filtered image 

is thresholded at each gray level. A Connected Components Algorithm partitions 
each of the thresholded images into potential object regions. The Super Slice 
Algorithm correlates perimeter points formed independently by the Non-Maximum 
Suppression and Connected Components Algorithms and a score is obtained for each 
gray scale threshold. These scores and several other algorithms form a set of 
Classification Logic. A short description of each algorithm follows. 

The Median Filter acts to extract the median value from a 5x5 pixel 
window and place that value in the center pixel location; the filter can be 
considered as a smoothing operator. 

The Gradient Operator is an edge detector which is defined as GRAD OP = 
max {|A-B], |C-D|} where A, B, C, and D each represent the sums of overlapping 


regions of 4x4 pixels each as seen in Figure A-2. The value of GRAD OP is 





Figure A-2. Gradient Operator 
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placed in the pixel location marked "X" which is one pixel to the left and 
above the center of the entire region. 

The Gradient Operator extracts edges in either the horizontal or vertical 
direction; the non-Maximum Suppression Algorithm then looks in a direction 
perpendicular to the edge for a larger gradient. If a larger value cannot 
be found, the edge under consideration is retained; the edge is removed if 
a larger value is found. The Algorithm is shown in Figure A-3. The gradient 
under consideration is a vertical one and the area examined for larger 
gradients is in the vertical direction. 

The purpose of the Connected Components algorithm is to segment the 
image data stream into smaller domains. Each small domain includes a single 
object in the image plane. This algorithm distinguishes between objects 


and isolates regions so that statistics for Classifiation Logic can be obtained. 


= If any x>y, y = 0 
y = Otherwise retain y 


= x, y are gradient values 


Figure A-3, Neighborhood for Non-Maximum Suppression 
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Assume that the original image has been thresholded and the result is 
in binary form with gray levels exceeding 9) shown as 1's in Figure A-4, Two 
lines are retained in memory so that each pixel can examine its neighbors to 

| the left and right and above and below. No diagonal connections are 
permitted under this convention and an adjacent (horizontal or vertical) 


pixel must be occupied in order to make a connection. No skips or gaps 





Figure A-4. Binary Image 


are allowed, and the computations start one pixel in from the edge. In 

Figure A-5 there are four distinct regions, A, B, C, and D. The only possible 
connection between regions B and C is through a diagonal, which is not 

allowed in the computations for the fourth row as seen in Figure A-6. Here, 
there is a connection between regions B and C and an equivalence statement, B = 


C, is carried along. At the end of the sixth row, there is another connection 
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Figure A-5, Computations for the Second Row 











Figure A-6, Computations for the Fourth Row 
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between C and D (C=D) and all the regions are completed as seen in Figure A-8. 


Figure A-7, Completed Image 


The areas of A, B, C, and D are computed by cumulating the number of pixels 
assigned to each. The perimeter is calculated by cummulating the number of 
pixels assigned to each region which are neighbors of zeroes. Each pixel has 
eight neighbors; a perimeter point is defined as having at least one zero 
neighbor, similarly an interior point is defined as having eight non-zero 
neighbors. 

The Super Slice Algorithm selects the appropriate threshold 
by matching the set of Non-Maximum Suppression outputs with the perimeter 
points of Connected Components for each threshold. Thus, each object may 
have a different threshold value than some other object within the same image. 

Another approach to segmentation to be explored under this contract 
is that of the Westinghouse Auto-Q system which is briefly described in the 
next paragraphs. Three types of data are extracted. The primary data are 


the straight-line contours of gray-level gradient. Thus, a line-drawing 





of the video image is generated. The second type of data are positional 


cues of gray-level closed objects (or "blobs"). The location of a blob 
generates a window within which recognition features are generated. The 

final set of data are statistical parameters computed during the preprocessing 
which may be used in texture classification. 

Operation of the preprocessor is on a line-by-line basis with 
respect to the input image. Therefore, video data may be handled directly. 
Furthermore, Storage requirements in the preprocessor are limited to single 
lines of data only. 


To generate the straight-line contours (subsets) of the image, it 


is necessary to first compute the two-dimensional gradient at each image point. 


This is done as shown in figure A-8 with a four-pixel window scanning 
across the image in a raster format. The gradient amplitude and angle are 
approximated as shown. The gradient direction is quantized to 16 discrete 
directions, as depicted in the diagram. To suppress the areas of negligible 
gradient activity (containing no significant contour or edge information), 

a threshold is applied to the gradient amplitude. 

After gradient thresholding the edges are generaily still too wide 
for subset generation. Therefore, a gradient thinning operation is performed. 
The operation basically "skeletonizes" adjacent colinear gradient directions 
to the peak or maximum points. 

The algorithm utilizes a raster scanning window containing a 
gradient cell "X" and 4 of its nearest neighbors. The scanning window is 
depicted at the top of figure A-9, The neighbors with colinear gradients 
are compared to "X". The largest gradient is then retained. This procedure 


is repeated sequentially for each gradient point in the image. 
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Figure A-8, Gradient Extraction Process 
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Figure A-9. Block Biagram - Blob Detector and 
Subset Generator - Operational Cycle 





Subset generation is accomplished by “growing” a line formed by 
adjacent parallel gradients. As before, a 5-cell scanning window is employed. 
The new data point is labeled cel] "X". Its four neighbors are examined 
(sequentially: A, B, C, then D) to find those containing a parallel 
(within a tolerance) gradient direction. If one is found, then "X" is added 
as the next point in the line from the neighbor. Neighbors that are colinear 
to the gradient of "X" are excluded to prevent false lines from forming. 

The operational cycle of the subset generator is diagrammed in figure 1-11. 

The blob detector detects the presence of a contiguous area of 
gray levels lighter (or darker) than its surrounding background. It 
operates independently of size, orientation, and position, and will detect 
all but. sharp, concave shapes. 

The operation of the blob detector is similar to that of the subset 
generator. The input data is the output of the gradient stage. Basically, 
the blob detector seeks to trace paths along contiguous, slowly changing 
gradient directions. Bookkeeping counters for each path being traced keep 
track of the gradient at the start of the path. When two paths from the 
same starting gradient join, a blob detection occurs. Additional bookkeeping 
counters measure the maximum and minimum X and Y excursions, providing a 
measure of the blob's size. 

Figure A-9 depicts the operational cycle of the blob detector. 


It uses ‘the basic 5-cell window scanning the gradient image. Each of the 


four (4) neighbors of the X-pixel is examined to determine if X should be 


added as the next point in a blob tracing path. The output of the blob 
detector consists of the blob polarity, center position, and horizontal 
and vertical dimensions. This data permits the object to be isolated for 


feature extraction. 








i The third preprocessor function is the collection of statistical 

. data for texture classification. The gray level image area is divided into 

| i. windows of 30 x 30 pixels for statistical data collection. The average gray 
level and average gradient amplitude is computed. A limited histogram of 
the gradients is accumulated; i.e., the number of pixels with gradient 
amplitude equal to zero, one, two, and three. Also, two additional 
parameters are computed: (1) the number of pixels with gray level > a, and 
(2), the number of pixels with gray level < b. The subset generator provides 
two statistics: (1) the number of subsets per window, and (2) the number 
of "long" subsets. 

The final processing of the data is accomplished in a programmable 
processor (general-purpose computer). Its task is to generate the recognition 
features and perform the target decision logic. A block diagram of the final 
processor is shown in figure A-10. 

1.3.2 Blobs and Groups 

To reduce storage and speed requirements and to reduce background 
interference, recognition features are not computed for the entire image. 
Instead, they are initially computed only within local rectangular areas 
whose positions are designated by the blob detector. Therefore, the blob 
detector in effect "cues" the processor to a local area containing a 
possible target. However, for those targets having complex shapes, such as 
aircraft, cues are also initiated by the presence of a "starter" subset. A 
starter subset is defined as one whose length exceeds a predetermined value 
(e.g., 1 > 5). For each starter subset within the image, a square area (or 
window) centered on the subset is also used as a positional cue for the 
processor. The blob and long subset windows are used to collect groups of 
subsets, as will be discussed later. 


As seen from figure A-~10 the first function performed by the processor 


is blob merging. Under certain conditions a single target can give rise to 
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Figure A-10. Final Processor 
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multiple (usually no more than two) blob detections that overlap. 


Therefore, the blob list in the preprocessor buffer stage is scanned for 
blobs with overlapping areas. Overlapping blobs are merged into a single 
new blob whose area will enclose the union of the original blbo areas. 


See figure A-1l1. 


Figure A-11. Blob Merging 


Following blob merging a search is made for several different 
"associations". In general, an "association" means that an element (e.q., 
blob) is within a specified distance from another element. An association 
of long subsets with other long subsets is a significant association. These 
pairings may later be screened to detect the presence of roads. Also the 
association of blobs with long subsets is examined. Subsequently, these 
long subsets are prevented from being used to collect a subset aqroup, since 


the blob is usually a more accurate cue. 





orm, 





When the associations have been made the process of group forming 
starts. Each blob or long subset defines a window. For each window, al] 
subsets are screened by X-Y position. All the subsets falling within the 
window are defined as the groun for that window. 

Further screening of the groups is done to eliminate subsets not 
belonging to a condidate target area. It should be noted that the gradients 
of the subsets belonging to any dark (light) object point inwards (outwards), 
with few exceptions. See figure A-12. 

Subset pairs with non~opposing (inconsistent) polarities 
do not usually belong to an object, but are merely 
background clutter. Therefore, long subset groups are screened of any 
subsets with polarities inconsistent with the long subset defining the 
group. Blob groups are screened of any subsets with polarities inconsistent 


with the blob color, and relative to its center. 





Figure A-12, Significance of Polarities Between Subsets 
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